Localizing Policy Gradient Estimates to Action Transitions

Function Approximation (FA) representations of the state-action value function Q have been proposed in order to reduce variance in performance gradients estimates, and thereby improve performance of Policy Gradient (PG) reinforcement learning in large continuous domains (e.g., the PIFA algorithm of Sutton et al. (in press)). We show empirically that although PIFA converges significantly faster than traditional PG algorithms such as REINFORCE which directly sample Q (without using FA), FA representations of Q are not necessary to reduce variance in performance gradient estimates, and PG algorithms which use selective direct samples of Q can converge orders of magnitude faster than PIFA. We present a new PG algorithm, called Action Transition Policy Gradient (ATPG), which uses direct samples of Q and restricts estimates of the gradient to coincide with action transitions, thus obtaining relative value estimates of executing actions, without using FA representations of Q. We prove that ATPG gives an unbiased estimate of the performance gradient, and converges to an optimal policy under piece-wise continuity conditions on the policy and the state-action value function. Further, in an experimental comparison with PIFA and REINFORCE, ATPG always outperforms both algorithms, taking orders of magnitude fewer iterations to converge on all but very simple problems.